Combining Cardiovascular and Pupil Features Using k-Nearest Neighbor Classifiers to Assess Task Demand, Social Context, and Sentence Accuracy During Listening.

In daily life, both acoustic factors and social context can affect listening effort investment. In laboratory settings, information about listening effort has been deduced from pupil and cardiovascular responses independently. The extent to which these measures can jointly predict listening-related factors is unknown. Here we combined pupil and cardiovascular features to predict acoustic and contextual aspects of speech perception. Data were collected from 29 adults (mean  =  64.6 years, SD  =  9.2) with hearing loss. Participants performed a speech perception task at two individualized signal-to-noise ratios (corresponding to 50% and 80% of sentences correct) and in two social contexts (the presence and absence of two observers). Seven features were extracted per trial: baseline pupil size, peak pupil dilation, mean pupil dilation, interbeat interval, blood volume pulse amplitude, pre-ejection period and pulse arrival time. These features were used to train k-nearest neighbor classifiers to predict task demand, social context and sentence accuracy. The k-fold cross validation on the group-level data revealed above-chance classification accuracies: task demand, 64.4%; social context, 78.3%; and sentence accuracy, 55.1%. However, classification accuracies diminished when the classifiers were trained and tested on data from different participants. Individually trained classifiers (one per participant) performed better than group-level classifiers: 71.7% (SD  =  10.2) for task demand, 88.0% (SD  =  7.5) for social context, and 60.0% (SD  =  13.1) for sentence accuracy. We demonstrated that classifiers trained on group-level physiological data to predict aspects of speech perception generalized poorly to novel participants. Individually calibrated classifiers hold more promise for future applications.

The Influence of Hearing Loss on the Pupil Response to Degraded Speech.

PURPOSE: Current evidence regarding the influence of hearing loss on the pupil response elicited by speech perception is inconsistent. This might be partially due to confounding effects of age. This study aimed to compare pupil responses in age-matched groups of normal-hearing (NH) and hard of hearing (HH) listeners during listening to speech. METHOD: We tested the baseline pupil size and mean and peak pupil dilation response of 17 NH participants (Mage = 46 years; age range: 20-62 years) and 17 HH participants (Mage = 45 years; age range: 20-63 years) who were pairwise matched on age and educational level. Participants performed three speech perception tasks at a 50% intelligibility level: noise-vocoded speech and speech masked with either stationary noise or interfering speech. They also listened to speech presented in quiet. RESULTS: Hearing loss was associated with poorer speech perception, except for noise-vocoded speech. In contrast to NH participants, performance of HH participants did not improve across trials for the interfering speech condition, and it decreased for speech in stationary noise. HH participants had a smaller mean pupil dilation in degraded speech conditions compared to NH participants, but not for speech in quiet. They also had a steeper decline in the baseline pupil size across trials. The baseline pupil size was smaller for noise-vocoded speech as compared to the other conditions. The normalized data showed an additional group effect on the baseline pupil response. CONCLUSIONS: Hearing loss is associated with a smaller pupil response and steeper decline in baseline pupil size during the perception of degraded speech. This suggests difficulties of the HH participants to sustain their effort investment and performance across the test session.

Copresence Was Found to Be Related to Some Pupil Measures in Persons With Hearing Loss While They Performed a Speech-in-Noise Task.

OBJECTIVES: To assess if a manipulation of copresence was related to speech-in-noise task performance, arousal, and effort of persons with hearing loss. Task-related arousal and effort were measured by means of pupillometry. DESIGN: Twenty-nine participants (mean age: 64.6 years) with hearing loss (4-frequency pure-tone average [4F-PTA] of 50.2 dB HL [SD = 8.9 dB] in the right ear and 51.3 dB HL [SD = 8.7 dB] in the left ear; averaged across 0.5, 1, 2, and 4 kHz) listened to and repeated spoken Danish sentences that were masked by four streams of continuous speech. Participants were presented with blocks of 20 sentences, during which copresence was manipulated by having participants do the task either alone or accompanied by two observers who were recruited from a similar age group. The task was presented at two difficulty levels, which was accomplished by fixing the signal-to-noise ratio of the speech and masker to match the thresholds at which participants were estimated to correctly repeat 50% (difficult) or 80% (easy) of the sentences in a block. Performance was assessed based on whether or not sentences were repeated correctly. Measures of pupil size (baseline pupil size [BPS], peak pupil dilation [PPD], and mean pupil dilation [MPD]) were used to index arousal and effort. Participants also completed ratings of subjective effort and stress after each block of sentences and a self-efficacy for listening-questionnaire. RESULTS: Task performance was not associated with copresence, but was found to be related to 4F-PTA. An increase in BPS was found for copresence conditions, compared to alone conditions. Furthermore, a post-hoc exploratory analysis revealed that the copresence conditions were associated with a significantly larger pupil size in the second half of the task-evoked pupil response (TEPR). No change in PPD or MPD did was detected between copresence and alone conditions. Self-efficacy, 4F-PTA, and age were not found to be related to the pupil data. Subjective ratings were sensitive to task difficulty but not copresence. CONCLUSION: Copresence was not found to be related to speech-in-noise performance, PPD, or MPD in persons with HL but was associated with an increase in arousal (as indicated by a larger BPS). This could be related to premobilization of effort and/or discomfort in response to the observers' presence. Furthermore, an exploratory analysis of the pupil data showed that copresence was associated with greater pupil dilations in the second half of the TEPR. This may indicate that participants invested more effort during the speech-in-noise task while in the presence of the observers, but that this increase in effort may not necessarily have been related to listening itself. Instead, other speech-in-noise task-related processes, such as preparing to respond, could have been influenced by copresence.

The pupil near response is short lasting and intact in virtual reality head mounted displays.

The pupil of the eye constricts when moving focus from an object further away to an object closer by. This is called the pupil near response, which typically occurs together with accommodation and vergence responses. When immersed in virtual reality mediated through a head-mounted display, this triad is disrupted by the vergence-accommodation conflict. However, it is not yet clear if the disruption also affects the pupil near response. Two experiments were performed to assess this. The first experiment had participants follow a target that first appeared at a far position and then moved to either a near position (far-to-near; FN) or to another far position (far-to-far; FF). The second experiment had participants follow a target that jumped between five positions, which was repeated at several distances. Experiment 1 showed a greater pupil constriction amplitude for FN trials, compared to FF trials, suggesting that the pupil near response is intact in head-mounted display mediated virtual reality. Experiment 2 did not find that average pupil dilation differed when fixating targets at different distances, suggesting that the pupil near response is transient and does not result in sustained pupil size changes.

Social observation increases the cardiovascular response of hearing-impaired listeners during a speech reception task.

Certain cardiovascular measures allow for distinction between sympathetic and parasympathetic nervous system activity. Applied during listening, these measures may provide a novel and complementary insight into listening effort. To date, few studies have implemented cardiovascular measures of listening effort and seldom have these included hearing-impaired participants. These studies have generally measured changes in cardiovascular parameters while manipulating environmental factors, such as listening difficulty. Yet, listening effort is also known to be moderated by individual factors, including the importance of performing successfully. In this study, we aimed to manipulate success importance by adding observers to the traditional laboratory set-up. Twenty-nine hearing-impaired participants performed a speech reception task both alone and in the presence of two observers. Auditory stimuli consisted of Danish Hearing in Noise Test (HINT) sentences masked by four-talker babble. Sentences were delivered at two individually adapted signal-to-noise ratios, corresponding to 50 and 80% of sentences correct. We measured change scores, relative to baseline, of pre-ejection period, two indices of heart rate variability, heart rate and blood pressure (systolic, diastolic, and mean arterial pressure). After each condition, participants rated their effort investment, stress, tendency to give up and preference to change the situation to improve audibility. A multivariate analysis revealed that cardiovascular reactivity increased in the presence of the observers, compared to when the task was performed alone. More specifically, systolic, diastolic, and mean arterial blood pressure increased while observed. Interestingly, participants' subjective ratings were sensitive only to intelligibility level, not the observation state. This study was the first to report results from a range of different cardiovascular variables measured from hearing-impaired participants during a speech reception task. Due to the timing of the observers' presence, we were not able to conclusively attribute these physiological changes to being task related. Therefore, instead of representing listening effort, we suggest that the increased cardiovascular response detected during observation reveals increased physiological stress associated with potential evaluation.

The Presence of Another Individual Influences Listening Effort, But Not Performance.

OBJECTIVES: The aim of this study was to modify a speech perception in noise test to assess whether the presence of another individual (copresence), relative to being alone, affected listening performance and effort expenditure. Furthermore, this study assessed if the effect of the other individual's presence on listening effort was influenced by the difficulty of the task and whether participants had to repeat the sentences they listened to or not. DESIGN: Thirty-four young, normal-hearing participants (mean age: 24.7 years) listened to spoken Dutch sentences that were masked with a stationary noise masker and presented through a loudspeaker. The participants alternated between repeating sentences (active condition) and not repeating sentences (passive condition). They did this either alone or together with another participant in the booth. When together, participants took turns repeating sentences. The speech-in-noise test was performed adaptively at three intelligibility levels (20%, 50%, and 80% sentences correct) in a block-wise fashion. During testing, pupil size was recorded as an objective outcome measure of listening effort. RESULTS: Lower speech intelligibility levels were associated with increased peak pupil dilation (PPDs) and doing the task in the presence of another individual (compared with doing it alone) significantly increased PPD. No interaction effect between intelligibility and copresence on PPD was found. The results suggested that the change of PPD between doing the task alone or together was especially apparent for people who started the experiment in the presence of another individual. Furthermore, PPD was significantly lower during passive listening, compared with active listening. Finally, it seemed that performance was unaffected by copresence. CONCLUSION: The increased PPDs during listening in the presence of another participant suggest that more effort was invested during the task. However, it seems that the additional effort did not result in a change of performance. This study showed that at least one aspect of the social context of a listening situation (in this case copresence) can affect listening effort, indicating that social context might be important to consider in future cognitive hearing research.